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The largest college within an online university of over 50,000 students invested significant resources 
in translating a complex assessment system focused on continuous improvement and national 
accreditation into an effective and efficient electronic portfolio (ePortfolio). The team building the 
system needed a model to address problems met throughout the planning, design, and 
implementation of the assessment and ePortfolio systems. The team adopted the FEAT model to 
ensure that multiple stakeholder perspectives were an integral component of how the assessment 
system and ePortfolio development worked together. This model consisted of four domains: 
functional encompassed how the software tool worked and was used; educational reflected the 
desired learning as a result of system implementation; administrative included policies and 
procedures, financial and human resources, and planning necessary for project implementation; and 
technical included the hardware, software, and networking infrastructure necessary for ePortfolio 
and assessment system implementation. The researchers documented the types of problems 
encountered in the process, the problem solvers involved, strategies used, and actions implemented. 
The researchers concluded that evaluating system development is more informative if a systemic 
approach is used to examine the interdependent relationships among the FEAT model domains that 
influence the overall system maturity. 


The university, a leader in distance education, has 
been preparing graduates in the field of education for 
over 40 years. This study was performed in the College 
of Education and Leadership, a college with over 
13,000 enrolled students in 12 different degree 
programs and over 40 specializations (Walden 
University, 2012a). Degrees range from a bachelor’s in 
educational studies to PhD programs in special 
education. Less than 1% of the total college enrollment 
is in teacher preparation programs, including the Master 
of Arts in Teaching (MAT), which is the primary 
environment used to prepare this study (Walden 
University, 2012b). The teacher preparation programs, 
including the MAT degree, are in early childhood 
education and special education. Also offered is an 
education specialist (EdS) degree in principal 
preparation. All programs lead to educator licensure in 
the State of Minnesota and are accredited by the 
National Council for the Accreditation of Teacher 
Education (NCATE). The programs are pursuing 
national recognition through the appropriate 
professional associations (Walden University, 2012c). 

In 2007, the College of Education and Leadership 
began developing licensure programs intended to 
certify teachers, and in 2008 decided to pursue national 
accreditation. The decision to pursue national 
accreditation sparked the need to examine the existing 
assessment system, emphasizing transition points and 
key program assessments to measure and validate 
program outcomes. A discourse followed that required 
the institution to define the components (e.g., transition 
points) of the assessment system. Transition points are 
the milestones that occur within a program, allowing all 
stakeholders to determine whether the candidate is 
meeting the expectations that lead to graduation. For 


the purpose of this study, the university students in 
teacher preparation programs will be referred to as 
candidates. Key program assessments, which are 
components of the transition points, are standards-based 
artifacts (e.g., tests, projects, papers) that demonstrate 
the candidate’s academic performance. Program 
outcomes and professional standards are used to guide 
what the candidate should know and do by program 
completion. Since the decision to pursue national 
accreditation, numerous concurrent projects related to 
the electronic portfolio (ePortfolio) were launched (e.g., 
implementation of new program assessments and 
transition points). The process to examine the 
assessment system also included due diligence to 
determine the best tool for the specific requirements of 
the college’s assessment system, which included the 
ePortfolio. The college selected the commercial 
product, TaskStream, as its ePortfolio and assessment 
management system. The web-based application, at the 
least, allows teacher candidates to submit assessments 
to their ePortfolio for scoring by trained faculty 
members using valid, reliable rubrics. TaskStream also 
allows faculty members to manipulate data and 
assessment personnel to run reports necessary for data- 
informed discussions. 

Figure 1 describes the general assessment process 
used by programs that lead to teacher licensure within 
the College (Walden University, 2010). The first step, 
preparation, includes the process of candidates moving 
through a structured curriculum that provides the 
defined knowledge, skills, and professional dispositions 
within a framework of transition points whereby 
progress is monitored and support, when needed, is 
provided. The institution’s curriculum and course 
designers use a rigorous program and course 
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Figure 1 
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development process to ensure that the curriculum 
effectively supports what is assessed and vice versa. 
Faculty evaluators receive training in the use of scoring 
rubrics, enhancing reliability within the process. The 
required training, developed in the institution’s learning 
management system (i.e., BlackBoard), is self-paced, 
customized by program and assessment, and facilitated 
by veteran ePortfolio users. The second step has two 
parts related to the assessments submitted by candidates 
and evaluated by faculty members. The first part, 
submission, includes candidates submitting assessments 
electronically to the faculty evaluator in their portfolio. 
The second part, evaluation, includes faculty evaluators 
providing detailed feedback to the candidates about 
their performance on assessments used to determine 
progress in course and program requirements. Feedback 
on all rubric criteria is not required; however, when 
feedback is provided to candidates, evaluators can copy 
and paste rubric language to clarify what is necessary to 
improve work from one performance level (e.g., 
acceptable) to another (e.g., target). Candidates have 
the opportunity to revise an assessment three times and 
resubmit for additional feedback and final approval. 
The third step, data collection, includes items collected 
in the ePortfolio, such as the assessments, standardized 
examinations, and field and clinical experience 
evaluations. Also used are follow-up surveys (e.g., 
candidate, exit, and alumni), grade point average, and 
course evaluations that are not part of the ePortfolio. 
The fourth step, data analysis, includes analysis and 
disaggregation from the college level to the program 


level. A synthesis of both quantitative data from major 
assessments and other sources and qualitative data from 
faculty members and other external groups is provided 
in reports generated by the assessment personnel using 
the ePortfolio software. The fifth step, data 
dissemination, includes disseminating data to 
appropriate groups at a defined time, thus allowing 
different groups to reflect on and engage in meaningful 
discussions about the results of data collection. Figure 2 
illustrates the movement of data and reports through the 
required councils and committees. The sixth step, use of 
data for program improvement, involves time allocated 
to allow the major stakeholders - including faculty, 
candidates, and school partners - to discuss the results 
of data collection on a systematic basis. Feedback on 
major assessments is solicited, and scoring rubrics and 
revisions of rubrics are made as necessary. 
Recommendations for changes at the program or unit 
levels are vetted through a defined process, including 
the formal governance structure. All changes are 
documented through the fonnal process, as defined by 
the institution. The use of a cyclical model allows data 
to be introduced, reviewed, and revisited as 
improvements are implemented and determined to be 
effective. 

Incorporating the ePortfolio into the college’s 
complex assessment system effectively and efficiently 
created several challenges. To meet accreditation 
expectations, the assessment system must include the 
technological capability to construct, implement, and 
maintain an ePortfolio to track individual candidate 
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Figure 2 

Data Preparation, Dissemination, and Discussion 
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learning. The system itself must use a technological 
vehicle to construct, implement, and maintain an 
ePortfolio for each candidate. The internal and external 
partners in solving this problem had to design and 
customize the ePortfolio to maximize positive candidate 
and faculty perception and intended use level. Specific 
barriers were overcome to operationalize the 
assessment system within a technological vehicle to 
harvest, organize, and format ePortfolio data. Barriers 
included the selection and implementation of a new 
ePortfolio system, the dismantling of a prior ePortfolio 
system that no longer met the needs of the institution, 
and rapid consensus-building amongst business and 
operational personnel who had little or no experience 
developing ePortfolio or assessment systems. 
Significant outreach to the partners who manage field 
experiences was necessary in order to design and 
customize the ePortfolio reports that would allow 
stakeholders to make data-based decisions and guide 
subsequent program improvement. 

In the following paper, we share our three-year 
journey to design, implement, and mature not only an 
assessment system, but also the ePortfolio template used to 
collect, organize, and report the data collected for program 
recognition and national accreditation. As such, our 
journey is one of balancing multiple institutional forces 
and voices, emphasizing many of the functional, 
educational, administrative, and technical problems that 
can arise in an initiative of this scope. Our journey is one 
to share, for it contains insight into how to handle the 
complex, sometimes competing, elements of successful 
assessment systems and ePortfolios. 

Literature Review 

The purpose of this article is to describe the lessons 
learned from the simultaneous development of an 


assessment system and corresponding ePortfolio 
template within a compacted time frame. Within the 
context of the literature review, the topics studied and 
reviewed include ePortfolios and assessment systems as 
well as a model for establishing stakeholder roles and 
perspectives. The literature review also serves as a 
critical step in this study, which utilizes a development 
methodology. 

In order to develop simultaneously an assessment 
system and a corresponding ePortfolio template, the 
assessment working group adopted the FEAT model 
(Robertson, 2006) to help balance multiple, competing 
forces within the stakeholder team. The FEAT model 
consists of four domains. The functional domain 
encompasses how the software tool works and how it is 
used (e.g., the application and its features). The 
educational domain reflects the desired learning as a 
result of implementing the system (e.g., teaching and 
learning). The administrative domain includes policies 
and procedures, financial and human resources, and 
planning necessary to implement the project (e.g., 
sustainable budgeting). Finally, the technical domain 
includes the hardware, software, and networking 
infrastructure necessary to implement the ePortfolio 
tool and the assessment system. Establishing FEAT 
domains ensures that multiple perspectives are an 
integral component of how the assessment system and 
ePortfolio development work together. 

ePortfolios serve a critical function by providing 
the data to inform the assessment system. The 
functionality of an ePortfolio system ideally reflects 
process and product (Anderson & Robins, 2006; Reeves 
& Okey, 1996). The process involves identifying which 
ePortfolio requirements inform various assessment 
system benchmarks. The product is what the candidate 
creates and uses to demonstrate candidate, program, 
and institution learning. For the institution, this may 
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include regional and professional accreditation 
(Brickley, Schwartz, & Suen, 2000; Carney, 2004; 
Clarke, 2009). In order for the process and the product 
to be mutually beneficial, certain criteria must be 
addressed. First, instruments used to evaluate candidate 
work must be varied, valid, and reliable (Wilkerson & 
Lang, 2003). Second, faculty evaluators must have the 
ability to assess the work quantitatively and 
qualitatively using such instruments as mbrics and 
checklists (Choban, 2004). Third, the ability to 
manipulate the data collected to report on specific 
groups or timeframes allows the institution to respond 
to specific audiences and needs (Barrett, 2001; Oner & 
Adadan, 2011). Balancing the complex needs of the 
ePortfolio and the assessment system often comes 
through differentiating the outcomes of the process and 
the product. The use of learning outcomes is a critical 
measure of success and involves applying the 
functionality of the ePortfolio tool in a manner that 
meets the educational needs of the candidate, program, 
and institution. 

ePortfolios containing candidate work and 
performance data assist in achieving the educational 
outcomes measured through the components of the 
assessment system. In this case, a clear purpose for the 
ePortfolio is critical (Barrett, 2001; Burke, Fogarty, & 
Belgrade, 1994, 1995) for pinpointing how the data 
from the ePortfolio are used to measure achievement of 
educational outcomes. This type of approach 
diminishes candidates’ ability to select their own 
artifacts, a common benefit of ePortfolios (Barrett, 
2001). However, pre-determining the artifacts makes 
the alignment between curriculum and assessment more 
thorough and coherent, which is similar to the 
curriculum vitae level of maturity described by Love, 
McKean, and Gathercoal (2004). The use of pre¬ 
determined artifacts solidifies a program’s adherence to 
professional standards (Dom, 2002; Ehrmann, 2004), 
which ideally are also aligned with the educational 
outcomes. Overall, a coherent design of the ePortfolio 
within the context of the assessment system allows the 
institution to determine whether an individual candidate 
or a related cohort have achieved the outcomes set forth 
by the program or institution. A common understanding 
of the context of the ePortfolio and assessment system 
must be shared among all stakeholders, including 
institutional administration, in order to realize fully the 
costs and benefits of both. 

Broad and deep administrative support is necessary 
to craft an assessment system that is informed by a 
robust ePortfolio. First, the executive leadership (e.g., 
Vice President, Chief Academic Officer) must establish 
a sustainable business plan (Jafari, 2004) that commits 
the resources necessary for both the ePortfolio and 
assessment system to thrive. Second, academic and 
operational team leadership (e.g., Program Directors, 


Assessment Systems and Electronic Portfolios 24 


Product Manager) must understand where and how the 
ePortfolio will be used to inform the assessment system 
and address rigorous accountability and accreditation 
measures (Clarke, 2009). Using this data productively 
for accreditation has an impact on the reputation and 
marketability of the institution (Levine, 2000). Third, 
academic leadership (e.g., Program Director) must hold 
faculty accountable for evaluating the candidate work 
in the ePortfolio, providing clear expectations and 
incentives for proper completion of the evaluation of 
candidate work without cither trivializing the process or 
making the workload too cumbersome for faculty 
(Strudler & Wexler, 2008). Overall, the complexity of 
our online organization has allowed executive 
leadership to allocate fiscal and human resources to 
tackle problems using the best-fit decisions in which the 
proper teams and individuals are leading the execution 
of specific plans. For example, assessment personnel 
(e.g., one Assessment Director and one Assessment 
Coordinator) focus on building assets in the ePortfolio 
tool while Information Technology (IT) staff are 
assigned technical tasks such as account roles and 
creation processes; rarely are the roles reversed. The 
relationship between the technical stakeholders and 
others relies on the ability of each role to fulfill its 
commitments and inform other stakeholders of the 
strengths and weaknesses of various technical 
strategies. 

Implementing ePortfolios within an assessment 
system requires complex technical decision making. 
Modern ePortfolio tools are maturing in sophistication. 
Their ability to disrupt the functioning of higher 
education classrooms (Christensen, 1997) depends on 
whether decision-makers can promote the variables that 
allow innovations to “stick” at the institutional level 
(Jafari, 2004). There are generally two types of 
ePortfolio tools, common tools and custom tools 
(Barrett, 2001). Common tools involve the use of 
everyday technology productivity tools, such as word 
processors, web page editors, and institutional 
homepage space to post static web pages that satisfy the 
need for an ePortfolio (Batson, 2002). Custom tools 
involve more sophisticated web-based database 
applications that may or may not be housed virtually at 
the institution. The database design of the custom tools 
allows for more robust privacy features, structured 
interactions with reviewers, and date-stamped feedback 
from portfolio viewers (Greenberg, 2003). With proper 
database design, the information can be harvested for 
use at the departmental or institutional level. Custom 
tools may require more institutional support and are 
generally more expensive. The authors recommend the 
use of custom tools because of the privacy and 
feedback features as well as large glossaries of 
standards (Truer & Johnson, 2003) that can be used to 
drive reporting capabilities. Batson (2002) specifically 
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refers to these types of tools as ePortfolios, which are 
also generally commercial software products with a 
global market requiring adherence to adaptability 
(Ittleson, 2001), flexibility, growth, and interoperability 
(IMS Global Learning Consortium, 2004). The 
institution selected the specific custom ePortfolio tool 
because it possessed superior privacy and 
interoperability along with robust reporting that can be 
adapted to inform specific requirements of the 
assessment system. 

The combination of the four domains of FEAT 
provides for a balanced and thoughtful approach to 
many of the problems faced by teams charged with 
determining a solution. Very few large, complex 
projects survive without administrative support and the 
technical resources to launch successfully. The authors 
found that even when a project does launch, 
sustainability is difficult if the functional relevance and 
educational outcome are weak. 

In order to create structure and ensure 
predictability as well as maximize efficiency, the 
authors adopted a developmental research approach. 
Richey, Klein, and Nelson (2003) described two types 
of developmental research in the area of media and 
technology. Type I research is intended to focus on a 
single instance of production, providing highly detailed 
descriptions of specific methods, including case studies. 
Type I research also emphasizes drawing conclusions 
based on context-specific models, analyzing the 
products and conditions. Type II research is intended to 
build knowledge and understanding of specific design 
processes rather than explore and explain a particular 
instance (as preferred in Type I). Type II developmental 
research commonly is used for model-building and 
includes survey research, observations, program 
evaluation, literature reviews, case studies, Delphi 
techniques, and think-aloud protocols (Richey et ah, 
2003; van den Akker, 1999). For the purpose of this 
study, Type I research will be used because of the 
emphasis on the design, development, and evaluation of 
a specific item (Richey et ah, 2003). 

Type I research is specific to contextual projects 
and designs. Specific elements of a Type I study may 
include the type of program being developed, the 
particular design processes used, the tools and 
techniques used in the process, and the context of the 
organization for which the program has been designed 
(Richey et ah, 2003). Specifically, this research 
examined the development of an assessment system and 
a complementary ePortfolio template. 

This research extends in multiple ways the 
scholarly conversation regarding ePortfolio 
implementation. First, the FEAT model provides a 
framework for establishing a balanced and 
knowledgeable implementation team. Second, the 
assessment process and data diagrams describe tangible 
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outcomes associated with collecting and discussing the 
data. Third, the methodology and timeline articulate 
specific actions and milestones necessary for successful 
implementation. Overall, the addition of this work in 
the scholarly conversation will help fellow adopters 
implement assessment systems and ePortfolios with 
greater clarity and collaboration. 

Methodology 

The design of this research represents formative 
approaches to the developmental methodology. The 
developmental method suggested by van den Akker and 
Plomp (1993) has a two-fold purpose. First, 
developmental research tends to support the 
development of prototypical products, including 
empirical evidence of effectiveness. In this study, the 
products include the assessment system and the 
ePortfolio template. Second, the developmental method 
tends to generate methodological directions for the 
evaluation of such products. In this study, the design 
teams met several times to evaluate the products as they 
evolved. This research involved studying the whole 
process and specific parts of the ideal assessment 
system and an ePortfolio used to inform it. The 
instructional components of this process and the 
subsequent products were designed as a result of the 
research. As a result of this study, new knowledge was 
created about the design of the assessment system and 
ePortfolio template, their development, and their 
evaluation (Richey et ah, 2003). The overall design of 
the research was to explore, explain, and design 
(Gibbons & Bunderson, 2004) an assessment system 
and ePortfolio template that is specific to the 
institution’s needs and outcomes. 

To develop the product described in this study, the 
following procedure (Figure 3 and Table 1) was used. 
First, a thorough literature review was conducted, 
including both professional and academic resources. 
The search primarily focused on assessment systems 
and processes, teacher education and its accreditation 
environment, and the design and content of ePortfolios. 
Institutional documents, such as the existing assessment 
plan, were used to identify and define the assessment 
system, including guiding principles, best practices, 
standards alignments, transition points and major 
assessments, support systems, and assessment 
resources. 

Second, a formative group of stakeholders was 
convened. This internal committee consisted of various 
academic, assessment, business, and leadership 
representatives (e.g., Program Directors, Field 
Experience Coordinators, and Assessment Director). 
These individuals were chosen because they 
represented the highest level of institutional intelligence 
about assessment systems and the ePortfolio. One of the 
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Figure 3 
Methodology 



Table 1 


Methodology Summary 


Step No. 

Title 

Outcome 

1 

Prepare Literature Review 

Determine existing research and institutional 
information. 

2 

Convene Formative Group 

Translate the existing requirements and guidelines 
into assessment system and ePortfolio. 

3 

Review Draft System and Template 

Provide feedback for revision. 

4 

Build Prototype Template and System 

Construct real and technological components. 

5 

Convene Summative Group 

Evaluate the revisions to the ePortfolio template. 

6 

Use Feedback to Revise Template and System 

Make changes based on feedback. 

7 

Test Template and System 

Determine if all components of template work 
properly. 

8 

Complete Final Review of Template and 
System 

Provide feedback for revision. 

9 

Revise Template and System 

Make changes based on feedback. 

10 

Launch Template and System 

Start using ePortfolio and collecting data. 

11 

Enroll Faculty and Candidates 

Add faculty as evaluators and candidates as 
ePortfolio owners. 

12 

Train Faculty, Staff, and Candidates 

Provide written and live instruction on the 
components and use of the systems. 

13 

Configure IT and Export Process and 

Schedule 

Ensure reliable data passage from one system to 
another. 

14 

Establish Assessment Schedule 

Determine calendar for when assessment data will 
be discussed. 
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roles of the formative committee was to translate the 
existing requirements and guidelines regarding the 
assessment system into an ePortfolio structure and 
operation. The commitment of the formative 
committee included weekly teleconference meetings 
and individual work time spanning a multi-year time 
frame. 

Third, an initial review of the draft assessment 
plan and ePortfolio template was performed by the 
formative group. A series of open-ended questions 
was used to determine whether or not the title, 
purpose, and order of the ePortfolio requirements 
matched the assessment system process. 

Fourth, a prototype ePortfolio template was 
developed based on input from various sources. Each 
requirement within the ePortfolio template was 
matched to an evaluation method, such as a rubric, 
and mapped to specific institutional, state, national, 
and/or professional standards. Each evaluation 
method also was assigned a frequency based on how 
often it needed to be reported. 

Fifth, the formative group was reconvened as a 
summative group to evaluate the revisions to the 
ePortfolio template. The role of the summative group 
was to provide feedback and/or approval of each 
individual requirement in the ePortfolio template. 

Sixth, a series of questions related to the content 
and design of the project was developed to guide the 
summative group in providing feedback. The 
questions solicited feedback on titles, structure, 
order, and formatting. The group also provided 
comments on the evaluation methods and standard 
alignments associated with the major assessments. 

Seventh, the summative group met via web- 
based conference to test the template with sample 
candidate and faculty evaluators. The sample 
candidate account was used to enter fictitious work, 
and the sample faculty account was used to evaluate 
the submissions. This step provided validation that 
all parts of the ePortfolio template were working 
properly. Sample reports also were generated to 
illustrate completion of assessment system 
requirements. 

Eighth, in order to complete the final review of 
the template and system, feedback from the 
summative group on both the ePortfolio template and 
the assessment system was analyzed and compiled. 
Comments specific to content were analyzed for 
pertinence to the development of the template and 
the system. Comments specific to design were 
analyzed for feasibility. The analysis of this data also 
was used to provide further refinement to the 
ePortfolio template and the assessment system. 

Ninth, to triangulate data from the committees 
and the institutional document review, the ePortfolio 
template was reviewed by the other members of the 
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assessment staff and by representation from the 
ePortfolio vendor. The other assessment staff 
provided feedback on comparability and 
interoperability with other institutional ePortfolio 
templates. The vendor representative provided 
guidance on efficient use of the product, including 
building individual data points that could be 
aggregated later for use with the entire assessment 
system. 

Tenth, revisions were made to the template 
based on input from the assessment staff and vendor. 
Pertinent content and feasible design revisions 
compiled in steps eight and nine were integrated into 
the template. A revised assessment system and final 
draft of the ePortfolio template were launched. 

Once the tenth step had been completed, a series 
of subsequent steps were launched related to 
supporting and extending the use of the assessment 
system and the ePortfolio. Eleventh, the assessment 
personnel were allowed to enroll faculty, staff, and 
candidates in the ePortfolio. Twelfth, the following 
stakeholders were provided training specific to their 
role in the process: 

• Candidate technical support staff were trained 
to solve or escalate technical questions. 

• Faculty were trained on the assessment system 
and the ePortfolio template, including 
evaluation methods (faculty members are not 
allowed to score assessments and be 
compensated without having completed the 
training). 

• Program directors and field experience staff 
were trained to use the system to check for 
submissions, evaluate transition point 
requirements, and generate reports. 

The authors noted that training ideally would 
occur before faculty and candidates gained access to 
the ePortfolio; however, the implementation timeline 
overlapped in such a manner that training prior to 
enrollment was not feasible. Also, the following 
tasks were performed as part of the launch of the 
ePortfolio: 

• Thirteenth, the IT team aligned each 
assessment placeholder with an associated 
requirement in the student (candidate) 
information system used to track progress on 
the degree audit. 

• Fourteenth, the assessment staff planned a 
detailed calendar of when assessment data 
would be harvested from the ePol6rtfolio, 
formatted into coherent data presentations, and 
disseminated to appropriate assessment 
committees for discussion. 
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Types of Problems 

Figure 4 describes four general types of problems 
that have occurred in the process of designing the 
assessment and corresponding ePortfolio. These 
problems have been classified according to the FEAT 
domains (Robertson, 2006). 

Functional 

All groups within the organization needed basic 
knowledge of the ePortfolio and its functionality. 
Translating a paper-based portfolio into an electronic one 
revealed several functional issues: scoring choices for 
evaluators using rubrics and design and use of forms used 
to track eligibility and performance at field experience sites. 

• Scoring choices for evaluators using rubrics: 
Software options and settings have the ability to 
exclude evaluators from awarding partial point 
scores. The system was designed to allow only 
whole number scoring choices offered to the 
evaluator in a pull-down menu. 
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• Design and use of forms used to track eligibility 
and performance at field experience sites: The 
original forms, which were very well-organized 
word processed documents, lost their complex 
formatting to achieve a simple, linear look and 
feel. Therefore, translating the form to the 
ePortfolio template took more time than expected. 
Some stakeholders also preferred the original 
word-processed version for aesthetic reasons. 

Educational 

A significant educational problem existed due to 
the numerous standards and outcomes that the 
assessments are used to measure. To address this 
problem, the stakeholders used functionality - called 
“tagging” - within the ePortfolio tool to manage all of 
the standards and outcomes. Tagging is the process of 
associating specific rubric criteria (rows) with a variety 
of standards and/or outcomes. Once a rubric row is 
tagged, it is possible to report the data associated with 
all rubric rows tagged to determine the degree to which 
a standard or outcome has been met by individual 
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Figure 4 
FEAT Diagram 
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candidates or cohorts. For example, major assessments 
in the program use a rubric criterion related to the skill 
of “selecting and using informal assessment.” The 
knowledge and skills related to this rubric criterion are 
aligned with various standards and outcomes. 
Numerous conditions influenced the decision to use 
tagging as a strategy to track how candidates performed 
at the skill of selecting and using informal assessment. 

• At the institutional level each criterion is 
aligned with program-level learning outcomes, 
college-level learning outcomes, and the 
college’s professional education unit outcomes 
that relate to assessment. 

• At the state level each criterion is aligned with 
Minnesota assessment-related standards 
including the Standards of Effective Practice, 
Core Standards for all Special Educators, 
Standards for Special Educators: Emotional 
Behavioral Disorders, Standards for Special 
Educators: Learning Disabilities, and the 
Standards for Early Childhood Educators. 

• At the national and professional levels, each 
criterion is aligned with assessment-related 
standards within the Council for Exceptional 
Children (CEC) Common Core, Emotional 
Behavioral Disorders, and Learning 
Disabilities, as well as the National 
Association for the Education of Young 
Children (NAEYC), and the National Council 
for the Accreditation of Teacher Education 
(NCATE) program standards. 

Administrative 

A significant administrative problem existed 
with the various levels of approval (i.e., IT, 
assessment, academic, and business) of a multi¬ 
layered, complex organization. Because of the highly 
integrated nature of the systems, even seemingly 
insignificant changes triggered a domino effect of 
system adjustments requiring the leadership from 
various teams to concur on changes before they 
happened. For example, for each ePortfolio there are 
various placeholders where assessments are 
submitted and evaluated. After an assessment is 
evaluated by a faculty member, the score is 
submitted to the administrative record. The content 
of the administrative record is exported from the 
ePortfolio system and imported into a table in the 
student (candidate) information system. This export- 
import process occurs on a weekly basis. After the 
information is imported, various staff members in the 
Registrar’s Office use the data to determine whether 
candidates have completed all of the requirements 
necessary to graduate (i.e., degree audit). The 
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following scenarios illustrate how system-level 
changes required a proactive approach from team- 
level leadership. 

• If any problems occurred with the export- 
import process, or if the process failed, 
degree conferral was impacted. Any 
problems with the import process required 
the directors of several academic, technical, 
and business teams to hold spontaneous 
meetings to resolve issues. 

• If the name of a placeholder or the source 
template undergoes name changes or other 
revisions, the linkages of the export-import 
process are broken and must be rebuilt. It was 
determined that name changes required a lock- 
step pattern of communicating any changes 
from academic to assessment to IT to business 
leaders to make system adjustments. In order 
to manage the process and reduce faulty data, 
the administrative stakeholders suggested and 
implemented a form to describe and manage 
changes. The use of this form mirrors the steps 
required to edit the template and adjust the 
export and import process, thus lending 
accountability to the entire process. 

Technical 

Significant technical problems needed the 
attention of the IT staff. For example, the integration 
of the student (candidate) information system and the 
ePortfolio allowed the data to be stored in the 
institutional data warehouse, thus allowing the data 
to be integrated into the institutional reporting 
scheme. Before the institution could rely on these 
reports for making institutional resource decisions, 
the IT staff had to design and build reliable reporting 
mechanisms. Members of the IT staff met with 
assessment and academic staff to determine reporting 
needs, including the demographic fields needed for 
filtering and disaggregating data. The outcome of the 
consultation was to create a library of reports related 
to specific aspects of the assessment process. A 
specific, critical report takes data from the ePortfolio 
and combines it with course rosters to determine 
which candidates have completed the assessment. 
This report had widespread positive impact for 
multiple reasons. 

• The information in the report allowed staff 
to monitor course sections and cohorts to 
determine the submission and completion 
rate for a given assessment. 

• Due to the nature of the institution, this 
report was necessary to track the work of 
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adjunct faculty who may or may not know 
all of their specific responsibilities. 

• Since the assessments are submitted during a 
specific course but are not necessarily a course 
requirement, this report allows academic and 
assessment personnel to track submission and 
cohort completion. 

• The programs range in size from dozens to 
thousands of candidates, so technical solutions 
that make data more accessible improve 
productivity. 

The four areas of FEAT are interdependent and 
often require a team of diverse professionals to resolve 
problems in order to reduce the chance that one 
particular lens—functional, educational, administrative, 
or technical—is exerting too much influence on the 
problem-solving process. The use of the FEAT domains 
allows the program leadership to balance the roles and 
perspectives of the teams assigned to solve specific 
problems in the development of the assessment system 
and ePortfolio template. 

Problem Solvers 

The decision to pursue accreditation and 
implement the assessment system and ePortfolio had 
broad representation throughout the university and 
college. Throughout the process of developing the 
assessment system and ePortfolio, the institution’s 
Project Management Office was charged with the task 
of making, managing, and monitoring the actions and 
outcomes of every team engaged in the accreditation 
process. In the functional domain, the Office of Field 
Experience collaborated on the development of the 
components of the ePortfolio related to what candidates 
experience in school classrooms and other types of 
educational settings. The Assessment Directors and 
Coordinators developed the ePortfolio template, 
participated in rubric development, and anticipated the 
types of reports needed from the ePortfolio. Faculty 
members participated in training and also engaged in 
rubric development and evaluation of candidate work. 
The Student (candidate) Support Team developed 
multiple resources used for training candidates to use 
the ePortfolio and answered numerous questions via 
telephone and web-based help. In the educational 
domain, the Program Directors provided leadership in 
the development of the transition points, key program 
assessments, and accompanying rubrics. In the 
administrative domain, the president of the university 
and vice president of the college provided direct (e.g., 
financial) and indirect (e.g., professional development) 
support necessary to implement the assessment system 
and develop the ePortfolio. The dean provided 
academic leadership and oversight on the accreditation 
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process. The associate deans facilitated working groups 
including academics, assessment, and administrative 
representatives. In the technical domain, the following 
was accomplished. Operational teams, such as IT, 
designed the infrastructure necessary for efficient 
account creation, etc. The vendor (TaskStream) 
provided technical and design training and consultation 
as needed. Overall, 500 candidates used the ePortfolio 
in licensure programs that certify teachers, and over 
13,000 candidates used it throughout the college. 

Problem Solving Strategies 

Once the team, consisting of a balanced 
representation of the FEAT domains, was assembled, 
the following strategies were used to address the 
problems identified as the assessment system and 
ePortfolio were developed. 

Collaborate to Find Solutions 

Collaboration was a primary method of solving 
problems. Many existing teams, such as the academic 
leadership for each program, were used as platforms to 
discuss problems and seek direction. New teams also 
were developed to respond to emerging needs. For 
example, the assessment team, which began as one 
individual, has grown to eight people to meet the 
demands of assessment, accreditation, and the 
ePortfolio. As new teams emerged, so did new methods 
of addressing problems collaboratively through shared 
responsibility. In many cases, when a problem arose, 
the owners formed a meeting to discuss and plan how to 
solve the problem. The appropriate actions were arrived 
at through consensus. The assessment team, assigned 
the task of developing the ePortfolio, constructed a 
prototype of how the ePortfolio would be configured to 
respond to the problem. Changes were tracked using 
internal forms. Then, multiple remote employees 
convened on a conference call to review the prototype 
and test its use. Various team members selected specific 
areas of the prototype to review in depth. Once all 
functionality of the prototype was revised and agreed 
upon, the assessment team built the active version in the 
live ePortfolio template. Other stakeholders then 
implemented plans to communicate the changes 
through coursework, e-mail, and announcements in the 
ePortfolio tool. 

Use Technology Effectively to Develop and 
Implement the ePortfolio 

Another method of working toward solutions has 
been the effective use of technology. For example, 
application sharing and web-based meetings have been 
used to conceptualize, complete, and review many of 
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the detailed steps in the development of the 
ePortfolio. For example, multiple stakeholders 
reviewed ePortfolio components to ensure that the 
design and content were ideal for the purpose. In 
many cases, the component of the ePortfolio was 
best reviewed by the office or team responsible for 
implementing the component. In the case of field 
experience applications and evaluations, the Office 
of Field Experience staff was charged with reviewing 
the functionality of the components. 

Pilot to Ensure Accuracy and Reliability 

A third method of working toward solutions, 
piloting, has been used to various degrees. Whether as 
proof of concept or trial-and-error testing, to ensure 
accuracy and reliability various members of the team 
have developed prototypes for the working parts of the 
ePortfolio. When a form or evaluation instrument was 
deployed, it was first built and tested in a controlled 
environment. In one case, a field experience placement 
form had to be translated from its complex word- 
processed format into a form used in the ePortfolio. 
Many stakeholders liked the organization of the word- 
processed form; however, it did not allow for easy, 
efficient aggregation, thus limiting the ability to make 
data-informed decisions about placements and 
supervision. Ultimately, the visual appeal of the word- 
processed document was forgone in favor of the simple 
ePortfolio form, which allowed for more efficient use 
of data. 

Respect Complexity to Understand and Rebuild the 
System or Process 

The final method, respect for complexity, has 
allowed the team to address problems by breaking 
down the components of the system or process and 
rebuilding it in a manner that works within the 
ePortfolio tool. For example, at one time the entire field 
placement process worked through word-processed 
forms delivered via e-mail. The team accepted the 
challenge of dismantling the complexity of the process 
and rebuilding the forms in the ePortfolio to create a 
new and innovative model for managing field 
placements. All of the aforementioned methods would 
not be as effective without a solid relationship with the 
ePortfolio vendor, where suggestions could be made 
freely in order to enhance the product and make the 
institutional operations more efficient. The vendor 
understood the complexity of the work being completed 
and made gradual improvements to address the needs. 
The use of practical meetings and team management 
strategies supports future steps in the process, including 
specific actions used to address problems and move the 
project closer to completion. 
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Actions 

As a result of implementing the steps in the 
methodology, the following actions were taken. Each 
action was categorized according to the FEAT domains 
in order to assess the balance of perspective and 
workload occurring with any given phase of the project. 
Table 2 summarizes the actions. 

Early in the process, proposed actions were 
conceptual in nature and balanced among FEAT 
domains. Between July 2009 and December 2009, the 
following tasks were accomplished. Most important, in 
the educational domain, the heart of the assessment 
system - the transition points and major assessments - 
were conceived, defined, and approved for each 
program. This work was led by the administrative 
action of hiring the first program assessment 
coordinator, who would eventually become an 
assessment director after an organizational change. 
These two tasks enabled the technical action of 
building, testing, publishing, and piloting the ePortfolio 
templates for all programs involved. Finally, once all of 
the educational, administrative, and technical tasks 
were completed, the assessment staff was able to begin 
preliminary faculty training on the transition points and 
major assessments. 

The next phase, spanning from January 2010 
through June 2010, involved finalizing the ePortfolio 
aspects of the assessment system for candidate work 
and data collection. All of this work was bolstered by 
the administrative action of hiring a program 
assessment coordinator to support the director. In the 
educational domain, the major assessments were 
written, edited, aligned with various standards, and 
approved for each program. These tasks enabled several 
technical actions. First, all forms used for field 
experience were developed, tested, and implemented in 
the ePortfolio. Second, all requirements in the 
ePortfolio templates were completed and reviewed for 
accuracy. Third, all components of the assessment 
system necessary for tracking candidate progress were 
mapped for regular exporting from the ePortfolio to the 
student (candidate) information system. Once all of the 
educational, administrative, and technical tasks were 
completed the assessment staff was able to begin 
continued faculty training on the major assessment 
rubrics, field experience forms, and the ePortfolio 
template layout. 

The following phase, spanning from July 2010 
through December 2010, involved the first candidate 
work and data collection cycles. All of the data 
collection work in this phase was used for the 
administrative action of submitting data reports for 
national accreditation. In the educational domain, the 
first major assessments were submitted by candidates to 
the ePortfolio and evaluated by faculty assessors; the 
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Table 2 

Timeline and Actions Organized by FEAT Domains 

Timeframe 

Major Tasks with FEAT References 

July 2009 - 

F: First faculty training 

December 2009 

E: Transition points defined 

E: Major assessments conceived and approved 

A: First assessment coordinator approved and hired (Later became assessment director) 

T: ePortfolio template built, tested, and piloted 

January 2010 - 

F: Second faculty training 

June 2010 

E: Major assessments and rubrics completed and approved 

A: Second assessment coordinator approved and hired 

T: Field evaluation forms built and tested 

T: Program ePortfolio built, piloted, tested, and approved 

T: Export to student information system mapped and tested 

July 2010 - 

F: Field evaluation faculty training and support 

December 2010 

F: First cohort field evaluation forms submitted 

E: First major assessment data harvested and discussed 

A: National recognition reports approved and submitted (SPA) 

T: Export to student information system implemented 

January 2011 — 

F: Field evaluation faculty training and support 

June 2011 

F: Second cohort field evaluation forms submitted 

E: All Major assessments submitted once each, data harvested and discussed 

A: National recognition reports approved and submitted (SPA) 

T: Export to student information system continues 

July 2011 — 

F: Field evaluation faculty training and support continues 

December 2011 

E: Refine ePortfolio requirements 

E: All major assessments submitted twice each, data harvested and discussed 

E: First program completers in new assessment system 

A: National recognition reports revised (SPA) 

T: Export to student information system continues 

January 2012 - 

F: Field evaluation faculty training and support continues 

June 2012 

E: Refine ePortfolio requirements 

E: All major assessments submitted three times each, data harvested and discussed 

E: Second program completers in new assessment system 

A: National recognition reports revised (SPA) 

A: Accreditation visit completed and recognition achieved 

T: Export to student information system continues 


Note. Robertson and Larkin (2011) 


data then were harvested by the assessment staff. The 
field forms also were submitted and harvested for the 
first time. Next, assessment staff members continued the 
functional tasks of faculty training to assess candidate 
work using the mbrics in valid and reliable ways. After 
the first submission of data was complete, the technical 
task of exporting data from the ePortfolio to the student 
(candidate) information system continued. 

As the implementation continued from January 
2011 through June 2011, the actions focused on 
refining the major assessments as they operate within 
the assessment system. All of the data collection work 
in this phase also was used for the administrative action 
of submitting data reports for national accreditation. In 
the educational domain, the major assessments were 


submitted by candidates to the ePortfolio and evaluated 
by faculty assessors; the data then were collected by the 
assessment staff for the second time. The field forms 
also were submitted and harvested for the second time. 
Next, assessment staff members focused the functional 
tasks of faculty training by collaborating with other 
offices to train field faculty members who assess 
candidate teaching in field. After the second submission 
of data was complete, the technical task of exporting 
data from the ePortfolio to the student (candidate) 
information system continued. 

From July 2011 through December 2011, the 
actions continued to focus on refining the assessment 
system and assessments the after the third cohort was 
complete. All of the data collection work in this phase 



Larkin and Robertson 


was also used for the administrative action of 
submitting data reports for national accreditation. In the 
educational domain, the major assessments were 
submitted by candidates to the ePortfolio and evaluated 
by faculty assessor; the data were harvested by the 
assessment staff for the third time. The field forms also 
were submitted and harvested for the third time. Next, 
assessment staff members focused the functional tasks 
of faculty training by collaborating with other offices to 
train field faculty members who assess candidate 
teaching in the field. After the third submission of data 
was complete, the technical task of exporting data from 
the ePortfolio to the student (candidate) information 
system continued. 

From January 2012 through June 2012, after the 
fourth cohort was complete, the actions continued to 
focus on refining the assessment system and 
assessments. All of the data collection work in this 
phase was also used for the administrative action of 
submitting data reports for national accreditation, which 
culminated in February 2012 with a successful site visit 
(notice provided as of April 2012). In the educational 
domain, the major assessments were submitted by 
candidates to the ePortfolio and evaluated by faculty 
assessors; the data then were harvested by the 
assessment staff for the fourth time. The field forms 
also were submitted and harvested for the fourth time. 
Next, assessment staff members focused the functional 
tasks of faculty training by collaborating with other 
offices to train field faculty members who assess 
candidate teaching in the field. After the fourth 
submission of data was complete, the technical task of 
exporting data from the ePortfolio to the student 
(candidate) information system continued. 

Findings 

The actions of the study led the researchers to the 
following findings about the simultaneous development 
of assessment and ePortfolio systems. They found that 
evaluating system development is more informative if a 
systemic approach is used to examine the FEAT model 
domains. The domains—functional development, 
educational connection, administrative support, and 
technical infrastructure established—have 

interdependent relationships that influence the overall 
maturity of the systems. 

As new tools were adopted, functional problems 
were treated as teaching opportunities. In this case, 
multiple layers of training needed to be addressed. First, 
the staff developing the assets with the ePortfolio 
needed skill development with the application. This 
included frequent web-based training with the product 
vendor as well as informal communities of practice for 
sharing new learning. Second, the faculty members 
needed training in three areas: the assessment system, 
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the ePortfolio tool, and the process of evaluating the 
assessment submissions. The training on the assessment 
system contained information about the process of 
assessment, key definitions, transition points, 
assessment requirements, and remediation plans. The 
training on the ePortfolio tool included authentication, 
navigation, locating candidate work, using rubrics and 
feedback mechanisms, and managing the revision 
process. Training on the process of evaluating the 
assessment submissions included describing each 
assessment, addressing all of the standards, analyzing 
comments and qualitative feedback, determining the 
rubric score, and submitting the evaluation. There also 
was a special section of the training that focused on 
eliminating bias, reducing ambiguity, and increasing 
accuracy and fairness in the scoring process. After 
completion of the training, the issues that surfaced 
included: 

• Compensation. Fifty dollars was provided for 
an intense training intended to last four hours. 
Comments were shared that many individuals 
did seven to eight hours of work and that one 
hundred dollars would have better represented 
the amount of time spent. 

• Ownership. The assessment personnel planned 
and delivered the first two training cycles; 
however, academic leadership determined that 
academic program directors would be the more 
ideal point persons. The third cycle of training 
would be led by the program directors, with 
consultation and guest facilitation from 
assessment personnel. 

• Corroboration. It was determined that more 
candidate samples were necessary to have a 
positive impact on any norming exercises. 

• Documentation. Academic leadership charged 
program directors with documenting which 
faculty have been trained and with assigning 
any and all evaluator tasks only to those who 
have been trained. 

Collecting data throughout the program ensures 
that tracking progress can occur objectively. Currently, 
one of the common notions of best practice in 
assessment includes using a variety of methods that are 
strategically distributed across the learning experience. 
This can be difficult in compressed programs or ones in 
which the conventional wisdom involves putting more 
assessments at the end of the program for fear of 
candidates not performing as well when they are less 
experienced with the knowledge and skills. In this case, 
transition point requirements were classified in three 
categories: administrative, academic, and field-based 
requirements. Once all of the requirements were 
juxtaposed in the ePortfolio template, a distribution 
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across all three categories was clearly demonstrated. 
However, most academic requirements were placed 
toward the end of the program, while many field-based 
requirements occurred earlier. Early in the program 
candidates are required to show fitness to be in the field 
(e.g., proof of professional liability insurance). Later in 
the program candidates need to demonstrate 
competency in field-based assessments (e.g., internship 
evaluation). 

Conclusions 

In this study, the ability to evaluate the assessment 
system as a whole is dependent on how the ePortfolio is 
designed. The assessment system, by describing the 
process and requirements that each candidate must 
complete, also identifies the benchmarks that determine 
program accountability. In order to track accurately 
candidate progress within and across cohorts, the 
ePortfolio template must reflect each and every 
requirement of the system. The risk of not representing 
each requirement may result in incomplete data or an 
inability to track candidate progress. The assessment 
system is built on transition points from admission to 
program completion. Each of the transition points 
contains multiple requirements, such as completing 
courses, passing assessment projects, or completing 
administrative forms. Each of those requirements must 
be translated to the ePortfolio design in order for such 
data to be available to program stakeholders. If 
requirements are ignored or combined, then data are 
missing or become more ambiguous. Ignoring or 
combining requirements makes the data more difficult 
to disaggregate, thus it becomes difficult to determine 
where program improvement is most needed. 

Once the assessment system and the ePortfolio are 
aligned and data are being collected, there will come a 
time when change is evidently needed. The data may 
come from a variety of sources such as surveys, rubrics, 
or other instruments. Once the data are organized and 
aggregated, importance should be placed on 
categorizing the data based on what the data suggest as 
potential improvements to the teaching and learning 
process. The categories may include the assessment 
itself, the course in which it is implemented, the 
academic program as a whole, the learning outcomes 
achieved by completing the program, and the 
operational actions of managing the program. In this 
case, data revealed that candidates had difficulty with 
rubric criteria related to applying concepts of valid and 
reliable assessment. After lengthy discussion, it was 
determined that the candidates who performed poorly 
on that particular component had not yet taken the 
elective research course, which most candidates save 
until the end of the program. Further discussion 
occurred as to whether this was an assessment, course, 
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or program problem. It was determined to be a course 
problem. The solution determined was to introduce just 
enough new content to one course in order for 
candidates to succeed at the specified rubric rows 
without overloading the course or removing the rubric 
criteria. Without the creation of an assessment system 
and an ePortfolio sophisticated enough to collect the 
data, this problem likely would not have been 
identified. 

When implementing the assessment system and 
collecting data through the ePortfolio, identifying 
efficiencies in data entry can save time and effort. One 
solution is to implement any data import features that 
the commercial ePortfolio tool may possess. For 
example, demographic data were imported in order to 
aid in disaggregation. Alternately, time and effort can 
be saved by using export features to benefit systems 
outside of the ePortfolio system, such as the student 
(candidate) information system. In this case, many 
candidates are required to take and pass a standardized 
teacher licensure examination to complete a transition 
point and the subsequent program. Once the test is 
taken and evaluated, the scores are sent from the test 
provider to the institution and then entered in the 
student (candidate) information system. The IT team, a 
partner in the technical domain, exports a file from the 
student (candidate) information system, importing it 
into the ePortfolio. At this point, faculty and program 
administrators can determine whether the candidate has 
taken the test and whether or not the score meets the 
minimum and the requirement has been met. This 
import allows hundreds of data points to be handled at 
once without the need for entering each one manually. 
More time can be spent on analysis and outreach to 
those who have not met the minimum score. Whether a 
candidate has or has not met all of the requirements 
necessary to move on to the next transition point can 
also be determined much more quickly. It is critical for 
the technical system to be developed for viewing data 
over both short and long periods of time in order to 
observe how the system is maturing. 

The assessment system and ePortfolio have been 
used for four academic semesters and have included 
hundreds of data points related to each candidate; the 
system is maturing in some ways, but not all. Much like 
human development, time alone is not an adequate 
indictor of maturity. While no specific definition exists, 
the authors consider the maturity of the assessment 
system to be dependent on various indicators that 
reflect maturity as an on-going process rather than a 
completed project or deliverable. None of the following 
indicators can be the sole indicator of maturity; rather, a 
balanced combination of multiple indicators is required. 
First, the authors value time or completed cycles as one 
critical indicator. The longer an assessment system has 
been in place, the more annual cycles over which it has 
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matured, although whether the activity within each 
cycle has been productive needs to be considered. 
Second, clear data points that can be expanded or 
collapsed are another indicator of assessment system 
maturity. The action of expanding and collapsing the 
data can help with drawing conclusions during analysis 
and discussion. Third, volume of data, including both 
overall quantity and breadth across the reporting unit, 
illustrates a commitment to collecting data. However, a 
commitment to collect data is not as significant as a 
maturity indicator as using the data is. Finally, regular 
and thorough use of the data provides opportunities to 
mature the assessment system. Maturity is stabilizing in 
terms of the quality and quantity of data collected; 
however, this is not a sole indicator of the maturity of 
the assessment system. More maturity is necessary 
regarding the data-informed discussions, actions as a 
result of data, revisions to courses and programs, and 
revisiting changes for efficacy. In reference to Figure 1, 
in order to mature the assessment, the cycle must be 
executed multiple times with special attention to 
efficacy: are the changes we are making having an 
effect? Or, when considering Jafari (2004), has data 
been used often enough—with proper attention to 
effectiveness—in order to eliminate the variables that 
cause changes not to “stick”? 

Next Steps 

Considering that much of this research is on the 
ideal design and content of the assessment system and 
the ePortfolio, future research should focus on the 
implementation of the system and its impact on 
candidate performance. In this case, prudent analysis 
would include determining whether early success on 
transition point requirements has any impact on success 
in later stages of the academic program. The analysis 
would include a comparative study between the rubric 
criteria for assessments used early in the program and 
rubric criteria for assessments later in the program, as 
well as program completion examinations used at the 
end of the program. Also, future study should be 
devoted to how revisions to the assessment system and 
ePortfolio are proposed, approved, and executed given 
the organizational structure and associated decision¬ 
making process. The future study would include an 
examination of the path followed by course and 
program changes proposed as a result of data discussed 
in various faculty and leadership meetings. Finally, the 
program will be reviewed by external consultants as 
part of the academic program review process to 
determine the maturity of the assessment system 
including the volume of data, process for dissemination 
and discussion, suggestion and tracking of action, 
quality and quantity of proposed changes, and impact of 
changes on subsequent assessments. The external 


Assessment Systems and Electronic Portfolios 35 


program review would include steps leading to the 
identification of areas for improvement within a 
maturing system. Then, stakeholders would brainstorm 
the most efficient actions to improve the system as a 
whole. 
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